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ABSTRACT 

A method pertaining to the identification and 
analysis of course objectives is discussed. A framework is developed 
by which post facto objectives can be determined and students' 
attainment of the objectives can be assessed. The method can also be 
used for examining the quality of instruction. Using this method, it 
is possible to determine mathematics course objectives and the 
attainment of these objectives by the group of students who passed 
the course. Subjects of the study were 816 first -year engineering 
students, enrolled in a university program during the 1970-1973 
period. The students were administered quarterly open-question 
examinations during the first three quarters of each year. The 
retrospective analysis data were collected in 1976. The exploratory 
charactei* of this research suggests that conclusions from the data 
must be considered merely as illustrations of the type of questions 
that can be analyzed within the proposed framework. (TJH) 
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1. Sunmaury 

In higher education, course objectives are seldc»n explicitly 

stated. Consequently, it is almost impossible to indicate 

which objectives have been attained by the students at the 

end of a course. Thus, there is often a need to establish, 

post facto, which objectives were pursued by the teachers 

and attained by the students. This need is especially important 

where a new instructional method is weighed against the usual 

on^ or where the identification of critical impact points 

for ifliproving an existing instructional method is the intent. 

An analysis of course examinations over several years ccin provide 

good insight about the objectives teachers have pursued, because 

the examination items can be considered operationalizations of 

the pursued objectives. And, by analyzing achievement on the 

examination problems, it is possible to determine which objectives 

the students have attained. 

In § 3 of this paper a method pretaining to objective Indentif ica :ion 
and analysis will be discussed. Thus a framework will be developed 
by which post facto objectives cein be determined and students* 
attainment of the objectives can be assessed. The method can also 
be used for examining the quality of instruction (see § 5). 
Using this method, we are able to determine mathematics course 
objectives and attainment of the objectives by the group of 
students who passed the course (see § 4). The results of this 
study are based on first-year engineering students in the period 
1970-1973, using "open question" examination formats. The explorative 
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ch2u:acter of this research suggests that conclusions from the 
data have to be considered illustrations of the type of questions 
that can be analysed within the proposed framework. 

2. Research-questions . 

In this paper, we discuss only a few of the research questions 
that could be discussed in the context of a retrospective cinalysis 
of implicit objectives. 

The central question is: what objectives were attained by the 
students, who passed the course? Several questions cein be 
distinguished form this general question: 

1. How can the pursued objectives be made explicit, so that they 
can be disaassed as well on content levels as on behavioral 
levels? 

2. What were the objectives (on both levels) pursued in the 
mathematics courses? 

3. What were the objectives (on both levels) the passed students 
attained? 

4. Were in the examinations the number of items, which are 
operationalizations of the objectives attained by the group of 
passed students, such a proportion of the respective examination, 
that based on the scores on these items a passmark could obtained? 
If not, is it possible to characterize the not-attained objectives, 
of which then must be concluded that the passed students have not 
or only partially attained them? 
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The relevance of these research questions comes from the situation 
that passed students (i,e. students with a pass mark) are considered 
to have attained course objectives, while often in practice evidence 
about this assumption is not considered. 

Ideally, questions pertaining to the attainment of objectives should 
be answered within a research design in which all the items frcan 
different examinations are connected by calibrating them on the 
same scale. For this it is necessary either to present some items to 
persons from different groups or to calibrate afterwards all the 
examination items on one group. In practice this requirement can not 
be met, so a retrospective analysis, in which plausible assumptions 
are made, is needed to answer the above questions. 

Method of research 

In this section we introduce the method of research which will be 
applied in the next section as a procedure by which the research 
questions can be answered. 

Considering the first question, we propose a two-dimensional grid 
fitting the situation of examinations based on implicit objectives 
(§ 3.1.). After having classified the examination items within 
the grid, question 2 can be answered (§ 4.4.). To address questions 
3 and 4, it is necessary to introduce in § 3.2. such constructs 
as: (1) item in an "open question" examination, (2) the mastery of 
an item by a student, and (3) measure for attainment of an objective. 



The method for conducting the retrospective analysis of course 
objectives is based upon the analysis of examinations results, 
which may be influenced by several sources of variation. These 
sources and some assumptions are discussed in--§ J. 3., the method 
of analysis in § 3.4. 

. Grid of objectives 
The items or problems in course examinations can be considered 
operationalizations of the pursued objectives. Each item is 
the operationalization of a particular objective, that may be 
described on two dimensions: (a) content and (b) behavior. 
Content refers to specific substantive elements in the body 
of knowledge. If the content levels are formulated too specif i- 
cally, items may be a composite of several objectives. Besides, 
then a too Icurge number of cells should be generated so that per 
ceil not a meaningful number of items will be obtsdned. 
Therefore, the content levels must be formulated broadly enough 
to enable an unambigious classification and a meemingful inter- 
pretation. This can be done by using general terms, such as chapter 
titles frcMtt text books or main topics frcxn the subject field. 
For the behavioral dimension, several classifications are 
available in the literature (e.g. the taxonomy of Bloom (1956) 
for the cognitive domain or its elaboration for mathematics by 
Wilson (1971). However, it is essential that the intellectual 
skill domain is as exact a classification of the measured 
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behaviors as possible. Since the analysis of objectives starts 
with examination problems, then the nature of these questions 
must be reflected in the behavioral levels. Therefore, we 
have chosen as a classification system the main categories 
of Polya (1957) , who distinghuishes between "problems to 
find" and "problems to prove". In each of the two categories 
some levels are defined, reflecting the different levels of 
difficulty of the examination problems (see § 4.2.) 
The cells of the grid and the row and columm totals should 
be studied as starting points of the analysis. Since examina- 
tions are samples of content and intellectual skills, it is 
much more likely that the cells of the grid will be representative 
of the course objectives if a number of examinations is aggregated. 
Then, not only a good impression of what objectives were pursued 
can be obtained from the grid, but also from the number of items 
per cell and per meurginal total an impression of the relative 
importance of the respective objectives. A caution should be 
noted. Empty or sparingly filled cells do not necessary mean 
that the objectives for the cells were not pursued; such cells 
may appear e.g. if there is some hierarchy within the coliims 
or the rows, so that some objectives are implicitly included in 
other objectives. 



3.2. Definitions 

An item or problem is the smallest unit in an examination to 
which a separate score will be given by the corrector and which 

I 
I 

8 



- 7 - 



can be solved independently fron the other problems in the 
examination. That means that so called 'pile '-questions (i.e. 
questions of which the solution is dependent on the results 
of one or more preceeding questions) will be considered as 
one item or problem. The m items in a cell are noted as 
(i = 1, 2, ... ,a) . 

Student j h as mastered item v. if his score on v. is more thein 
^ ^ 

a certain fraction of the maximum score S^; in other words if 

his score S. . on v. applies to: S. . $ aS. with 0 $ a $ 1. 
ID 1 ij 1 

In every application of the method, the value of the parameter 
a iTiUSt be chosen. This choice will depend on 'local' factors 
like the character of the objectives of a particular course 
or judgements about the appropriate level of mastery. 

Attainment of an objective will be defined with the help of 

the definition of mastery of an item by single students. 

By introducing the parameter a, the score of a student on an 

item is dichotomized. This can be made explicit by stat:ng a 

1 it S^j/S^ ^ a and a o if the reverse holds. 

A measure of attainment of an objective operationalized in the 



number of students with a 1 on Vj^ 
total number of students on v. 



item v^ is the ratio: 
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3.3. Assumptions and sources of variation . 

Students differ in their capability for mathematics. If we indicate 
with 0 the continuum underlying the ability of students to solve 
mathematics problems , then every student has a por ition on 
this continuum, and it is meaningful to assume a probability 
distribution g(0) for the group students. 

The quantity p^, introduced above, is an estimation based upon 
one item of the stochastic varieible tt , the true measure of 
attainment of the objective represented by the cell of the grid 
from which the item is drawn. 

Consider a cell of the grid with items v^ with p^ (1=1, 2, m) . 

Generally, the p^ (i = 1 , 2, . . . , m) in a cell will differ. Apart 
from influences by chance, this should not be the case because all 
the items in the cell are operationalizations of one objective. 
Iii5)ortant questions are: (a) how do we explain this variation and 
(b) how do we combine the information from all the items in a cell 
and to interpret this on one scale as a measure for the attainment 
of the objective represented by that cell. Three sources of veuriation 
can be considered: (a) students, (b) instruction and (c) items. 
Students differ in ability, preparation, attitude, etc. The 
quality of instruction is influenced by different teachers 
personalitids, teaching-styles, instructional procedures, etc. 
Since these two types of variation can be considerable and will 
influence the shape of g(6) it is important to aggregate different 
course examinations where as little variation has occurred as 
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possible. We assiune that the combined influence of both sources 
is constant over the considered period of time and that g(0) 
has the standard normal distribution ^(0) (Assumption 1) . The 
first paurt of this assumption is plausible in situations in 
which, over the period of time during which a course will be 
studied, nothing heis been changed with respect to (a) the 
entry reqxiirements , (b) the preceeding instruction, (c) the 
group of executive teachers, (d) the instructional procedures, 
(e) the basic textbook, etc. 

In the third source of variation, items may differ in dif ificulty and 
use of enabling skills, etc. Moreover, in the situation of the 
retrospective analysis, the items come from different examinations, 
taken by different groups of students (from the same population) . 
Considering the practice of aggregating examinations, it follows 
from the first assumption that differences in p-values in a cell 
are due to differences between items. Thus, for a cell of the 
grid the p^(i= 1, 2, m) not only pertain to a measure of 

attainment of the objective as represented in the respective 
items v^ (i = 1 , 2, m) , but these values also reflect 

differences in the difficulty of the items. 

Psychometric properties of an item v^ can be described by the 
item characteristic curve, i.e. the chance P^(l|0) to master v^ 
given the ability level 0 (see figure 1). It seems not unreasonable 
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Insert fugure 1 about here (see p&ge 32) 



,^rd, iy70) to assume that the item charactei f c curves have 
the normal ogive form, i.e. they have the mathematical properties 
of a normal cumulative distribution ( Assumption 2) . See for a 
discussion of the normal ogive model Lrrd & Novick (1968). 
The normal ogive item cheuracteristic curve of v. has a point 
of inflection at 9 = b^ (see figure 2). At this point of the 0- 
scale the probability of a correct auiswer is .50; b. is called 
the difficulty of the item v 



Insert figure 2 about here (see page 32) 



Lord and Novick (1968, p. 375} show that from assumptions 1 and 
2 follows that b^^ i/P^# with p^= biseral correlation between 
the dichotcxDlzed itemscore and 0, while is defined by 

+ 00 

\ = / * (0) d0 

in which the stochastic vcuriable tt^ is estimated by p^ (i= 1^ 2, 
. . . , m) ; P| is called the discriminating power of item v^. 
For all the items within the analysis, we assume that the 
discriminating power is the same, i.e. p^=p ( Assumption 3) . 
By this, p has become a scale factor which can be left out of 
consideration. Thus, we may write: b »z., in which z is 



estimated via the above formula using p^. 

In we have for each item v^ a measure of difficulty on the 

same quantitative scale 9. Earlier in this section, we saw that the 
values of the respective items v^(i-l, 2, m) not only are 

an estimation of a measure of attainment of the objective represented 
by each v^, but also reflect the differences in difficulty of the 
respective items. Now, we can not only interpret z^(i=l, 2, m) , 

estimated via p^, as a measure of difficulty for item v^, but we 
can also take the line that the pertains to the attainment of the 
objective operationalized in the respective items v^. This result 
will be used in the next section. 

Assunqption 3 can be tested by correcting, for each item v^, the 
item-test correlation for attenuation and spuriour^^iess emd checking 
if these are the same for all items. However, thxs is not possible 
in the situation of a retrospective analysis because the students 
did not take all the items # since different examinations were taken 
by different students. 

3.4, Measure for the attainment of objectives 

In this section, measures for the attainment of objectives by the 
students will be discussed. We will discuss the cases of cells with 
only one, respectively at least two items. In each case the criterion 
for attainment of the objective has to be chosen; this is a matter 
of rational judgement. 
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a. Cells with 1 item . 

At the end of § 3. 2. a measure p of attainment for an objective 
operationalized in an iten v was introduced. In this case, it 
is easily to define (without needing the assiimptions of § 3.3.): 
the students have attained the course objective represented 
by item v if p ^ 3 with 0 $ 3 $ 1 • For example we may judge 
that a given objective is attained if at least 75% of the 
students have reached thie chosen level a of mastery of an item 
(thus B = .75) • 



b. Cells with more than 1 item . 

Before discussing the two approaches, we shall define several 
notations with respect to a particular cell of the grid: 

- items : v, , v , v^ 

I z. m 

* number of students (generally 

different for each item) : , , • • • , N 

12 m 

* nxamber of students that 

master an item : n, , n. , n 

1 z. m 

* measure of attainment of 

an item; p^="i/N^ : p^, p^t ...f p^^ 

LI. Referring to the definition of p^ of each item v^ and to 

the case of a cell with 1 item, we can consider all the 

items which apply p^ ^ $. Let us denote by x the nximber of items 

from v., v^, V for which the students have attained the 

12 m 

chosen level of mastery a for each item. We define as follows 

I 
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The group of students has attained the objective if — ^ y , 
which 0 g Y g 1 * 

The choice of Y is again by judgement • By choosing this 
definition of attainment, of the assumptions in §3.3. we 
only need the first part of assiimption 1, viz. that the 
combined influence of the sources of variation (a) students 
and (b) instruction is constant over the considered period 
of time. 



Note that in this definition only the number of problems 
by which an objective is operationalized is considered. It 
is possible that the value of m could be very small, by which 
the application of this measure is not very meaningful. 

X 

Exaxnple: If m^l, then ^ can be only 0 or 1. If ma 2, then 

* can be only 0, ^ or 1. 
m 

By taking into account only the number of items, all items 

have equal weight in the procedvire. Use is not made of the 

fact that different numbers of students are performing on 

the respective items. Thus, the score of a single student 

performing on an item taken by a small number of students has 

a relatively greater weight than in the case of an item taken 

by a greater number of students. This will not be a problem 

if. (a) all the items are of almost the same difficulty (i.e. 

the values p., p^, p are almost equal) or (b) the 

\ c m 
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nuxabers of students taking the respective items are almost 
equal. 

In educaticmal practice those conditions are not always 
fulfilled. An example will help to illustrate the point. 
Let an objective be operationalized in three examinations 
in three different items and let the number of students in 
these examinations be 300 r 400 and 40 respectively. This is a 
realistic situation in ovir university. Moreover, Jet us 
ar^ume that only one item is attained (p^3) by the group 
of students. If the attained item is the one made by the 
group of 400 students and the two not-attained items are 
those in the examinations with the smallest nuaaber of students 
than, considering the whole population of students it may be 
reasonable to regard the item presented to the large group 
as a more representative operationalization of the objective 
than the two other items. 

From this point of view, it is desirable to look for a procedure 
in which not only the difficulty of the items, but also the 
number of students taking the items will considered. 

b2. To arrive at a procedure for which the two conditions are 

fulfilled the assumptions of § 3.3. are needed. There, we saw 
that for each item v^ (i- 1, 2, m) in a cell we have a 

estimated by p. via the transformation 
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where ^(9) is the standard normal distribution. The z^ (i» 
1, 2, ...,!&} can be found in a table of the standard normal 
distribution. It was concluded that z^ is not only a measure 
of difficulty of item v^, but also refers to the attainment 
of the objective operationalized by v^. 
By confuting for each cell 



the different numbers of students per item are taken into 
account. Moreover, the weighted mean z* of difficulties can 
be considered as representing the average d;Lfficulty of the 
items about the objective students are expected to attain. 
By means of the inverse transformation at each z*, a value 
p can be found. This p can be interpreted as a sort of 
'mean' value of the p^'s for the items in a cell. Note, 
that p* , as the transformed of the weighted mean of z^'s, 
is not the same as the weighted mean of the p^'s. 
For each objective, corresponding to a cell of the grid, 
p* can be interpreted as a measure of the attainment of the 
objective. Therefore, we define: 

The group of stiyients has attained the objective if p* > g, 
with 0 ( & f I. 
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Consistent with other parameters, the choice of g is a 
matter of judgement. 

Note, that the case of one item per cell is a special 
case (viz m= 1) of this* 

Application 

. Available data * 
We have analyzed the mathematics courses for first year 
engineering students at our university in the first three 
quarters of 1970/71, 1971/72 and 1972/73. Of each course the 
examinations at the end of the quarter, immediately after the 
teaching period, and at the end of the course year eure 
cinalyzed. 

In § 2 we pointed out that in this analysis the central 
question is: what objectives were in fact attained by the 
group of students which passed the examinations. Thus the 
method to determine the attainment of objectives was applied 
to this specific group. 

The data for this retrospective analysis were collected in 
1976. It appeared, that not every teacher of mathematics 
classes had saved the list with scores of the students on 
the respective problems. So, the analysis could only be 
applied on a part of the students that took the examinations. 
Table 1 contains the number of available data per examination. 

.1.9 
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Each examination V!;s labeled with a number. The first digit 
indicates the queurter of the course yeax in which the 
mathematics course was given. The second digit indicates 
the several examinations of the particuleu: course. 



Insert table 1 about here (see page 33) 

From table 1 we see that the availcd^le data on some examinations 
were very small. Tnerefore, the analysis of what objectives were 
pursued is based upon all the examinations, but the analysis 
of what objectives were attained is only applied on those examinations 
with at least 15 passed students. 

In § 4.5., the measure for attainment of objectives discussed in 
§ 3.4. under b2 is applied. With respect to the assumptions 
discussed in § 3.3., the following can be Scdd. Assumptions 2 and 
3 are underlying the analysis, their fulfilment was presupposed. 
The first part of assiamption 1 is plausible for the considered 
mathematics courses in the period 1970-1973, because in this 
period there were no changes in the entry requirements, the 
preceeding instruction (secondary school) , the group of executive 
teachers, the instructional procedures (lectures and practice 
sessions) the basic textbook, etc. Finally it was assiimed for 
the group of passed stuu its that the variable 0, expressing 
the cdDility to solve mathematics problems has a standard normal 
distribution. 

From these considerations we conclude that the results of 
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the analysis will have no general validity. They must be 
considered mainly as illustrations of the possibilities of 
the method. 

4.2. Grid of objectives . 

The problems in the examinations where distributed over the 
following content categories ; 

1. Mathematical induction ; 

2. Limits (of. sequences; functions of one variable: continuity, 
differentiability, computing of limits, theorem of I'Hopital); 

3. Differential calculus (finding the derivative, linear 
apprc:;imation, mean value theorem, extrema, points of inflection); 

4. Integral calculus (primitive function, Riemann sum, definite 
integral, length of arc, area and volume of surfaces of 
revolution) ; 

5. F\inctions of two variables (continuity, differentiability, 
partial derivatives, linear approximation); 

6. Differential equations (different types of first order equations); 

7. Series (tests for convergence and divergence, power series, 
series expansions of functions, Taylor's formula, computation 
with series) ; 

8. Improper integrals (tests for convergence and divergence) 
These categories reflect the main topics in the three 
mathematics courses. The categories 1 , 2 and 3 refer to the first 
quarter course: mathematics I , 4, 5 and 6 to mathematics Hand 7 
and 8 to mathematics III. The number of categories is such that each 
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item couid be categorized uiiambigiously according to content, 
while yet each category contains a meaningful number of items. 

Indicated in § 3.1. is the classification in levels of behaviour 
using the main categories of Poyla (1957) , who distinguishes 
between "problems to find" (F) and "prc^lems to prove" (P) . 
Additionally a rest category, "reproduction" (R) , was used. 

F. Problems to find 

The items in this category are characterized by "canpute" 
"determine", "find" and/ or "approximate". Three levels have 
been distinguished: 

Fl: elementary problems, solved with only one standard method 

(i.e. a for the students well known method); 
F2: problems solved by using several standard methods after 

each other or by using a standard method after a non-trivial 

re-formulation of the problem; 
F3: problems solved with non-standard methods (i.e. new problems). 

P. Problems to solve 

These items are characterized by words as: "prove" or "examine 
if •.."• We distinguish four levels: 

Pi: problems indicating which definition or theorem must be 
applied and/ or which standeurd type of proof can be used; 
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P2: problems wherein the theorem to be used is not given and/ 

or a non-trivial re-formulation of the problem is necesseury 

as a first step; 
P3: problems in which several definitions, theorems and/ or 

standard types of proof must be used (i.e. new problems); 
P4: proofs of generalizations or specializations of well-known 

theorems, of which it can be expected that the students have 

not learned them by heart. 

R. Reproduction 

This category is not further subdivided. 

4.3. Categorization of problems 

To indicate how we have interpreted both type of categories of 
the preceeding section, we offer some examples in this section. 
It appeaured that it was not always easy to put the items in a 
behavioral category. Sometimes it was difficult to distinguish 
between the categories Fi and PI; e.g. the differences between 
problems to prove and problems to find on 7 series fade away 
because the use of a test for convergence or divergence is in 
fact the computation of a standard limit. Or: finding an interval 
of convergence of a power series is in fact fully analogous to 
examining the convergence of a series with fixed terms. 
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4.3.1. Examples of problems to find . 



Pi: integral calculus; p = .85 



Find: 



1+x 



x^(l-x) 



dx. 



F2: differential equations; p = .63 
Solve: 



dy 2 2 
^dx " ^ + x+ 1. 



Fl: series; p = .86 

Find the interval cf convergence of 



nx 



n=2 



In X 



F2: limits; p = .76 

Given the function f with 



f (x) = r- , x>0. 

1+ln X 



Define f in the point 0 so that f has right hand continuity 
in that point. 

F2: differential calculus; p = .64 
Given the function f with 

2 1 2 
f (x) + jln (x+1) . 
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Find the extrema, points of inflection and asymptotes; find the 
intervals on which f is increasing and decreasing and also the 
intervals on which the graph of f is concave upward and concave 
downward (using the second derivative method) . Sketch the graph 
of f . 
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4.3.2. Examples of problems to prove > 

Pi: functions of two variables; p = 0.33 

Prove: ^ ^ 

2 2 

lim = 0 

x-K) X +y 
y-K) 

PI: series; p = .86. 

Examine the following series for (absolute) convergence or 

divergence: ^ ^ 

^ n +3n+5 



2 

n=l 2n +4n-l 



P2: limits; p = .25. 

Given the function f with 



- - . -X In X 

f (x) = X , X > 0 



= 0 , X 0 

Prove that 

lim f (x) = 0; 

find the right hand derivative in 0 

P2. series; p = .25. 

The n-th term of a series is given by 

n+1 , 
n X +1 



Prove the convergence of 



1 1 . 

n=0 " 



P4: differential calculus; p = .35. 

A function f is three times continuous differentiable with f(0) =1, 
f(l) «2 and f'(0) «1. Prove, using the mean value theorem, that 
there is a point a with 0<a<l such that f"(a)«0. 
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4.4. Pursued course objectives 

The result of categorizing the examination items within the 
grid is summeurized in table 2 

Insert table 2 about here (see page 34) 



By analyzing information in this table, some important 
conclusions pertaining the pursued objectives can be derived: 

a. Apparently, in examinations the mathematics teacners laid 
stress on an operationalization of the objectives mainly on 
levels Fl, F2 and Pi and, to less extent, P2. The remaining 
categories ("new problems") did not or seldc»n appear. 

b. The grading of mathematics examinations uses compensatoric 
model, i.e. a student gets a pass mark if his examination 
scora is at least 55% of the maximum score. From table 2 
we <;oncluded that a student could receive a pass mark if 
he disregarded preparation for problems on the level of 
P2, P3 and P4. This means that while higher level problems 
were inde iiscussed during the lectures and discussion 
groups, the department of mathematics accepted that students 
needed only take cognizance of the higher level objectives, 
but that they did not need to master them. 

c. Less attention was paid to "problems to prove" then to "problems 
to find". Exceptions were the content categories 7 "series" and 
8 "improper integrals". But in section 4.3. we have pointed out 
the difficulty with the categorization of the problems on series; 



since the relationshio between tha problems on improper 
integrals and the indicated problems on series i the same 
held to the category improper integrals, 
d. The row totals reflect the relative importance of the 
respective content categories. Two comments have to be 
made: (a) category 1: mathematical induction was indeed a 
small topic, only treated on the level PI and (b) the 
numbers in categories 7 and 8 were based upon four examinations, 
the remedning upon three examinations. 

4.5. Attednment of course objectives 

In t£U3le 3 the computations according to the method b2 in § j.4. 
are svmmarized. The results are based on a portion of the 
examinations (see table 1). The raw data are presented in the 
appendix. 

Before discussing conclusions from an cinalysis of table 3, some 
parameters have to be discussed. 

The choice of the parameter a for mastery of a problem (see 
3.2.) by a studeat was a =.66. The rationale behind this choice 
is the following. Most of the essay problems in the examinations 
had a meucimum score between 2 and 5 points. A student who made 
a minor error, e.g. in the computation or in the wording of the 
solution, would not get the maximiim score. However, we could 
assume he/she understood the solution of the probleJi. 
In this case the score on a problem was often 1^ out of 2 points 
2 out of 3 points, 3 out of 4 points, etc. i.e. a score of at 
least .66 of the maximum score. 

P7 
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In § 3 •4, we discuss two measures to express whether th-^ group of stud Tits 
has attained its goals or not. For reasons explained in § 3.4. 
we will choose p , defined in that seccion, as measure for the 
attainment of the goals. This measure expresses how well the 
group of students, in this paper the group of passed students, 
has attained the course objectives. 

When have the passed students attained the objectives? From 
tcdDle 2 we have concluded, that in the mathematics courses 
primarily objectives on the levels Fl, F2 and Pi were pursued, 
and, to a less extent, level P2. The other levels were so scarcely 
represented in the examinations that earlier on is concluded that 
those leve"'s were not pursued. Restricting ourselves, therefore, 
to the levels Fl, F2, Pi and P2, we chose as criterion for 
attainment of an objective $ = .75. Thus, if in a cell p* .75 was 
recorded we said that the objective was attained by the group of 
passed students. 

Now, from table 3 scxne important conclusions were drawn pertaining 
to the attainment of the objectives by the group of passed students. 



Insert table 3 about here on separate page (see page 35 



a. All the objectives, but one (viz. (3, F2)), which were 

attained by the group of passed students (p ^ .75) appeared 
to be of the type Fl: elementcury standard problems to find. 
Looking at the columns, only tne group of objectives Fl was 
attained by the students; although one cell was clearly not 
attained. 
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b. None of the objectives on the level of PI and P2 was 
attained, although some cells were close to the criterion. 

c. Because the row totals in table 3 were computed over different 
behavioral levels, it was not useful to apply the criterion 

of attainment of an objective on the row totals* From the 
last columm of table 3 can be seen that there were poor results 
on 1. mathematical induction, 2. limits and 5. functions of 
two veuriables. 

d. Because the Fl-type problems form usually less thein half the 
number of items of an examination an interesting conclusion 
could be drawn with respect to the considered group of students. 
Although the passed students performed well on the attained 
problems of type Fl, their score on these problems was 

usually not sufficient for getting a pass mark on the examinarion. 
To get the pass mark on the examination of the course every 
student has to master several of the objectives of which we 
concluded that they wer a'jc attained by the whole group of 
passed students. This means that teachers of following courses 
must take into account the fact that the group of students 
had attained only a relatively small part of the course 
objectives for courses which axe prerequisites for his/here 
courses. This means that appeurently it is the practice of 
mathematics education within the Depsurtment of Mathematics to 
accept passed students who may not have attained most of the 
course objectives but only taken cognizance of them. 




2 



9 
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5. Discussio ^ 

The procedure for a retrospective analysis of course objectives 
is not a test in a statistical sence. It is meant as an attempt 
to bring sooie order in an unsurveyable amount of exaiuination 
results, so that discussions on these results will be possible. 
Then^ discussions on the objectives of courses can be conducted 
using eurguments which are based more upon objective data then 
personal feelings cuid impressions of teachers. 
In the preceedingr we have discussed the questions: (a) which 
objectives were pursued in the mathematics courses and (b) which 
of them were attained by the students with a pass meirk. The answers 
to the first question are important for improving the considered 
mathematics courses. The emswers to the second question are 
impcrtcuit in connection with the construction of followiug courses 
of which the concerned course is one of the prerequisites. With 
this procedure we can indicate a general knowledge base upon 
which teachers jf following courses may build upon. 
We have considered the passed students of every examination. No 
distinction was made between examinations immediately after the 
teaching of the course and examinations later on in the yeeir 
(a second opportunity") , because the students who failed the 
examinations immediately after the course were provided an extra 
opportunity. Therefore, in our choice of the subgroup of students, 
the ''passed students", no one was counted twice. 
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Some concluding remarks and points for further research have to 
be mentioned. 

1. Under the assumptions prt^sented in § 3.3. per cell, the 

measure of attainment p* of an objective is computed in § 3.4. 

via the transformation z*-^p* . The transformations are carried • 

i 1 

out via the ctimulative standeurd normal distribution. From a 
table of this distribution can be seen that p varies alsmost 
proportionally with z outside the trials of the distrib^^tion. 
Thus, to get a quick impression of the results the weighted 
meeuis of p^-values can be computed, as long as the P^*^ not 
too small or too large. 

2. The number of items per cell will influence the accuracy of the 
estimation by p* of the true value of the measure of attainment 
of an objective. How to bring this element into the method 
discussed in § 3.4. has to be studied. 

3. The method presented in this paper can also be applied to 
examine the quality of instruction. Then, the measures of 
attainment of an objective have to be interpreted as measures 
for the quality of instruction. The examinations immediately 
after teaching of the course have to be analyzed for all the 
students who took them. 
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Appendix 

This appendix contains the data upon which the retrospective analysis 
of the attainment of objectives is conducted. The data are grouped 
per content category. The colums are consisting of: 

a. the behavioral levels Fl, PI, etc. 

b. per item v^ the number of students that passed the examination 
to which the item belongs. 

c. per item v^ the number n^ of students that masters the resp. items 
From these data table 3 can be derived: the number of items can be 
counted, per item p^ = '^i/N^ can be computed, etc. 



1. Mathematical 
Induction 

a b c 

PI 40 9 
17 11 
13 8 
28 16 

2 . Limits 

a b c 


Limits (cont.) 

a b c 

Pi 28 15 
17 6 
28 17 
28 5 


Differential 
Calculus (cont.) 

a b c 


Integral 
Calculus (cont.) 

a b c 


Fl 17 13 
(contO 28 27 
17 15 


Fl 53 44 
(cont.) 18 15 
18 13 
18 13 
18 8 
18 16 
53 45 
53 49 
18 15 


P2 17 3 
40 10 


F2 17 12 
28 26 
40 38 
40 21 
17 14 
28 18 


Fl 40 14 
17 8 
28 9 
17 7 
17 13 
28 22 


P3 17 0 
P4 17 4 


3. Differential 
Calculus 

a b c 


PI 28 12 

P4 40 14 
17 8 


F2 53 11 
53 31 
53 24 
18 14 
18 11 
78 5 
53 29 
18 18 


F2 17 11 
28 15 
17 13 
14 19 
17 9 
17 8 


40 38 
17 14 
17 11 
17 14 
17 11 
17 16 


4. Integral 
Calculus 
a b c 

Fl 18 14 
14 6 
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5. Functions of 
two variables 



Integral 
Calculus (cont.) 



PI 



18 
18 
18 



15 
5 
10 



Differential 
Equations (cont.) 



F2 



53 
18 
18 
18 



Pi 18 



a 


b 


c n 


Fl 


18 


15 


PI 


18 


5 


P2 


18 


6 1 




18 


3 1 



7. Series 



Fl 



6. Differential 
Equations 

a b c 



Fl 



18 

53 



15 

36 



39 
12 
16 
10 



b 


c 


50 


44 


38 


27 


87 


75 


87 


55 


87 


72 


72 


9 


50 


44 


50 


33 


38 


31 


36 


33 


38 


36 



Series 
(cont. ) 



Fl 50 
(cont.) 87 
36 



F2 



50 
42 



Pi 



50 
38 
87 
38 
87 
50 
50 
38 
38 
87 



P2 



87 
87 
50 



42 
75 
21 



10 
10 



31 
33 
54 
30 
75 
41 
35 
29 
12 
62 



62 
52 
35 



Series 
(cont. ) 



P4 



50 
38 
38 



8. Improper 
Integrals 



Fl 38 



Pi 



87 
50 
50 
50 
38 



P2 



50 
38 



10 
16 
13 



37 



61 
47 
29 
21 
18 
28 



21 
6 



34 



32 - 



1.00 !> 




Fig. 1: Item characteristic curves 
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Fig. 2: Normal ogive item characteristic curves. 
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Tabel 1; Number of students per examination 



exam, 
number 


11 12 13 14 15 16 21 22 23 24 25 26 31 32 33 34 35 36 37 


number of 
students 


66 21 40 16 46 4 85 12 37 10 46 21 107 4 38 23 104 31 55 


passed 
students 


40 17 17 12 28 3 53 5 18 8 18 7 87 3 10 13 50 14 38 



These examinations are used in the analysis of attained objectives. 



Table 2 : Number of items per objective. 





Problems to find 


Problems to prove 


Reprod. 


1 

Total 


Fl 


F2 


F3 


Itotal 
F 


PI 


P2 


P3 


P4 


Total 
P 


|1« mathematical induction 










4 








4 


— 


4 


2. limits 


9 


11 




20 


7 


4 


1 


1 


13 


3 


36 


3. differential calculus 


16 


10 




26 


1 


1 




4 


6 


— 


32 


4. integral calculus 


18 


15 


_ 


33 


3 








3 




36 


5. functions of two 
variables 


6 






6 


2 


6 






8 




14 


8. differential equations 


7 


7 




14 


1 








1 




15 


7. series 


28 


7 




35 


20 


10 


1 


4 


35 




70 


8. improper integrals 


2 


1 




3 


10 


2 






12 




15 

222 


Total 


86 


51 




137 


48 


23 


2 


9 


82 





"si 



Table 3 ; Results of the retrospective analysis 







Fl 


F2 


PI 


P2 


P3 


P4 


1 Total 3) 










m 


P* 


m 


P* 


m 


P* 


m 


P* 


m 




m 


P» 


1 . 


mathematical induction 










A 
H 


44 














1 4 


44 


2. 


limits 


6 


.50 


6 


.55 


4 


.41 


2 


.23 


1 


.00 


1 


.24 


18 


.46 


3. 


differential calculus 


9 


.88 


6 


.80 


1 


.43 




— 




.00 


2 


.39 


16 


.82 


4. 


integral calculus 


11 


.82 


8 


.56 


3 


.57 














22 


.69 


5. 


functions of two 
variables 


1 


.83 


- 


- 


1 


.28 


2 


.24 


- 


- 


- 


- 


4 


.40 


c 


differential equations 


2 


.72 


4 


.73 


1 


.59 














7 


.69 


7. 


series 


14 


.77 


2 


.22 


10 


.73 


3 


.67 










29 


.71 


8. 


in^rpper integrals 


1 


.97 






6 


.68 


2 


.10 










9 


.(7 


Total 


45 


.78 


26 


.60 1 


30 


.65 


9 


.48 















1) m is the nvunber of problems in which the objective is operationalized. 

2) p* the measure for attainment of the objectives by the group of passed students (see § 3.4.) 

3) only based upon the Fl, F2, Pi and P2 columns. 
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